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Abstract 

Humans explore the world around us using our able sens- 
es. Numerous studies have suggested that the greater the 
number of sensory modalities stimulated at any one time, 
the richer our experiences will be, in that the experiences 
may have a higher level of impact and be more memorable 
across a variety of our senses. (e.g. Bahrick & Lickliter, 2000; 
Spence,2002; Stein & Meredith, 1993). This is becoming in- 
creasingly important and relevant as we enter the Al-gener- 
ated virtual worlds where our visual perception should have 
a certain resemblance to our analog world experiences. If 
human perception is a cross-experiential phenomenon, how 
could we enrich artificial intelligence with multisensory cre- 
ative abilities to support the same processes and outcomes 
for material interaction? 


The ability to perceive correspondences across sensory mo- 
dalities resembles the phenomenon of synaesthesia (Cytow- 
ic, 1989). Synesthesia is the combination of several senses 
simultaneously. It is an unusual condition which gives rise to 
a merging of the senses. For example, smells may trigger the 
experience of shapes, or letters may give rise to the percep- 
tual experience of colour or may cause the experience of any 
combination of tastes, smells, shapes or sensations (Simner, 
2012). These misunderstandings may be why the phenom- 
enon was considered unreliable for over a century after its 
discovery by Francis Galton (1883; Sachs 1812). In the last 
decade, we can trace an increased interest in cross-sensory 
experiences signalling synaesthesia as its correlated phe- 
nomena (Dixon, Smilek, Cudahy, & Merikle, 2001; Mattingley, 
Rich, & Bradshaw, 2005; Ramachandran & Hubbard, 2001a, 
2003). While for most, mental imagery may be most vivid in 
the case of visual images (Kosslyn, 1994), it is of keen inter- 
est to note that rich mental images can be generated via the 
sense of smell, touch, taste, and sound (e.g. Klatzky, Lederman 
and Matula, 1991; Reisberg, 1992; Stevenson and Case, 2005). 


Through the intersection of two disciplines, User Experience 
Design (UXD) and Color and Materials Design (CMD), we aim 
to explore the possibilities of employing an intelligent agent 
to generate 3D materials based on merged multisensory in- 
puts rather than relying on existing algorithms, visual styles, 


and previously constructed patterns. The exploration will be 
in the form of an experiment and workshop conducted with 
a trans-disciplinary group of graduate students from the 
User Experience Design and Colour and Material Design dis- 
ciplines. We will use customised text to engage a 3D scene Al 
generator trained with available sensory data sets enriched 
with multi-perceptive input from the workshop participants. 
Our key novelty is the use of artificial intelligence trained with 
multi-sensory material properties to generate Al-generated 
3D materials for virtual and potentially hybrid products and 
environments. 
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Introduction 

In this paper, we aim to investigate opportunities for involving 
artificial intelligence in the design process of creating 3D vir- 
tual worlds. Our focus will be on the Al generation of 3D digi- 
tal materials and their potential to arouse sufficient sensory 
and emotional stimuli. Several studies have suggested that 
the greater the number of sensory modalities stimulated at 
any one time, the richer our experiences will be, in that the 
experiences may have a higher level of impact and be more 
memorable across a variety of our senses. (e.g. Nikolic & Rus- 
so, 2019; Bahrick & Lickliter, 2000; Spence,2002; Stein & Mer- 
edith, 1993). Accordingly, In the Al-generated virtual worlds, 
our sensory and cognitive expectations lead us to search for 
resemblance from our analogue world experiences. Hence, if 
human perception is a cross-experiential phenomenon, how 
could we enrich artificial intelligence with multisensory cre- 
ative abilities to support the same processes and outcomes 
for material interaction? 


Through the intersection of two disciplines, User Experience 
Design (UXD) and Color and Materials Design (CMD), we aim 
to explore the possibilities of employing an intelligent agent 
to generate 3D virtual materials based on merged multisen- 
sory inputs rather than relying on existing algorithms, visual 
styles, and previously constructed patterns. 


VIRTUAL SKIN: CO-CREATING 3D MATERIALS WITH SYNESTHETIC ARTIFICIAL INTELLIGENCE 


Our approach is hybrid and includes several methods applied 
in different research stages. In the first phase, we conducted 
a single case experiment, which we will describe in detail and 
present the findings. This led us to the second phase, where 
we prototyped and experimented with the Al-driven text-to- 
image platform for generating and rendering virtual materials 
on 3D objects. The whole process and the framework used 
are described in this paper's Synesthetic Artificial Intelligence 
Generation chapter. Lastly, we plan to conduct contextual in- 
terviews in a workshop with a mixed group of students se- 
lected from the User Experience Design (UXD) and Color and 
Materials Design (CMD) majors. 


Our key innovation is using artificial intelligence trained with 
multi-sensory material properties to generate Al-generated 
3D virtual materials for meta - and potentially hybrid - prod- 
ucts and environments. 


Background 

In our conceptual approach, synesthesia plays an important 
role in designing multi-sensory meta-perception beyond the 
physical world. Synesthesia is a psychological phenomenon 
related to the human brain’s capability to combine several 
senses simultaneously. In particular, stimuli on one particular 
physical sensory can produce additional sensory experiences 
for which sensory inputs do not exist (van Leeuwen, Singer 
& Nikolié, 2015). It is an unusual condition that gives rise to 
a merging of the senses. For example, smells may trigger the 
experience of shapes, or letters may give rise to the percep- 
tual experience of colour or may cause the experience of any 
combination of tastes, smells, shapes, or sensations (Simn- 
er, 2021). One of the most reported types of synesthesia is 
audio-visual (AVS)(Afra, Anderson, Funke et al, 2012), where 
sound stimuli can evoke the visual experience and vice versa. 
In particular, sounds such as the middle note 'C’ can induce a 
red colour experience, but the same note three octaves high- 
er can become green ( Ginsberg, 1923). Likewise, we can have 
different varieties of sense responses to sensory stimulation, 
such as; the taste of a particular food (gustatory) can associ- 
ate us with the visual appearance of the food (Cytowic, 1989); 
when we hear a particular sound (auditory) can induce smell 
of a specific food (olfactory) (Beeli, Esslen & Jancke, 2006). 
Therefore, despite early scepticism, contemporary research 
in the field proves that synesthesia exists. 


Furthermore, the interaction with materials in regard to a 
product, space, or environment is essential for a completely 
holistic multi-sensory user experience. Hence, from the per- 
spective of colour and material design, “texture is an essen- 
tial component of materiality that evokes the sense of tactil- 
ity,” often translated through a flat, 2D image that is applied 
onto the surface of a 3D object to define its colour, gloss, 
transparency, roughness, and other key physical attributes 
(Lefteri, 2014). A texture in 3D rendering engines like Photo- 
shop, Substance Painter, Substance Designer, Blender, and 
Keyshot require inputs whose details must be specified un- 
ambiguously. These tools make use of generating textures in 
a procedural material workflow. (Jantunen, 2017). Procedural 
materials are created using mathematical models instead of 
pre-stored data, reducing the storage needed for these mate- 
rials and thus creating texture maps to be applied to models 
in an actual runtime. Dong et al., 2019). 


As an example case study, Procedural Material Generation 
(Jantunen, 2017) delves into the steps to create a procedur- 
al material in five steps with a focus on different material 
properties such as texture, patterns, micro-surfaces, colour, 
and lastly, defines the procedural aspect of the material to 
identify the properties which can be changed while still re- 
taining the essence of the material. Our workflow (Figure 1) 
combines the aspects of synesthetic modalities with pro- 
cedural material workflows to generate multi-sensorial ma- 
terials, unlike a 3D rendering engine, whose inputs must be 
specified unambiguously and in complete detail, without fur- 
ther sensory extension when it comes to no explicitly stat- 
ed properties (Ramesh, et al, 2021). The design of products 
and experiences are typically the result of many fine-meshed 
and complex processes (Jongerius, 2010); and our proposed 
workflow uses the sensory nature of synesthetic procedural- 
ity when designing multi-sensory meta-perceptions. 


This multi-sensory, layered approach is based on an archival 
collection of information sourced through 1) scent, 2) sound, 
3) emotion, 4) taste, and 5) movement. The sensory experi- 
ences are categorised as layers that, when combined togeth- 
er, compose a unique material outcome. 


ARCHIVE OF INFORMATION 


LAYER 1 LAYER 2 LAYER 3| LAYER LAYER 5) 
SCENT SOUND EMOTION TASTE MOVEMENT 


Figure 1. Multi-sensory layered approach 


During the first phase, digital prototypes of 3D virtual materi- 
als were created using Open Al's text-image generation Al sys- 
tem DALL.E. Over 150 unique samples were created through 
prompts that described various material attributes regarding 
colour, texture, finish, and form. For the experiment, fifteen 
samples of visual distinction were short-listed (Figure 2). 


Figure 2. Virtual material samples created on DALL.E with prompts (left to right) beige 
green exploding fuzzy fluidic sphere, black white eternal colliding amorphous sphere, 
and high gloss stretchy blob slime beige sphere 


Experiment 

A single-case experiment research was employed to initiate 
the first phase of the 3D virtual materials generation. We in- 
vited six graduate students to participate in the experiment 
from two graduate design programs, User Experience Design 
(three participants) and Color and Materials Design (three 
participants). The participants included three females, two 
males, and one non-binary, of American, Indian and Mexican 
origins, and in an age range of 23-27 years old. 


The experiment was conducted in a controlled lab environ- 
ment with a defined layout, ambient temperature, sound, 
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light, and projected image control (Figure 3). The Al-gener- 
ated digital material samples were displayed on the screen 
for a period of three minutes, during which the participants 
observed the sample and noted their sensorial responses in a 
provided worksheet. (Table 1). The experiment lasted 45 min- 
utes, followed by 15 minutes of guided discussion. 


Figure 3. Single-case experiment set-up 


Findings 

Upon the completion of the first phase, participants re- 
vealed directives toward sensory modalities in regard to a 
series of Al-generated materials. The results from the exper- 
iment revealed the ability of the participants to look beyond 
the aesthetic attributes of the virtual materials. With this 
in consideration, it should be noted that half of the partici- 
pants stated had it not been for the structure of the work- 
sheet, they would not have related the visual with all of the 
senses. Four out of six participants stated that they found 
associations to be made quicker with certain senses than 
others. The sensorial responses experienced by the partici- 
pants were then used as inputs to generate the synesthetic 
Al-generated virtual skin. 


Table 1. Example of worksheet and responses from the experiment 


Touch Smell Taste Sound Visual Emoti- 
ons 
Sam- Tender Soft Delicate Lullaby Baby's When 
ple 1 Soft scent sweet- skin pressed 
ofa ness will 
—, mother release 
<2 the flow 
of milk 
Sam- Fuzzy Wet Fresh Soft Aplanet Awalk 
ple2 Soft grass tasteof breeze withun- through 
bloo- known paddy 
3 ming biome fields 
‘ee spring inthe 
time morning 


Synesthetic Artificial Intelligence Generation 

Starting from text descriptions, we used DALL-E to gener- 
ate the image of a sphere texturized according to the initial 
prompt - as demonstrated in the single-case experiment — 
and then we extracted the corresponding texture. The pro- 
cedure to fit the bump map and the texture was then refined 
using the Stable Diffusion model to progressively refine the 
starting template mesh, which in our case, is a sphere. 


A neural network generates an RGB texture and predicts the 
magnitude of the movement along the normal direction for 
each vertex. To compute the displacement, we encode the 
position of the vertex x E RS using Multiresolution Hash En- 
coding (MHE). We parsed the features obtained h E€ R™* toa 


SirenNet, an MLP (MultiLayer Perceptron) with sinusoidal ac- 
tivation function: 


x =x +n -d-SirenNet(MHE(x )) 


where n € R"! are the vertices normal and d the maximum 
displacement. The texture is computed in a similar way by 
mapping the positions of the pixels to the range [-1,+1], 


RGB, = SirenNet (sin sin ((2i/(H-1)-1) 72), sin sin (( 2j/(W-1)-1)z1)) 


where H, W are the texture dimensions. This initial transforma- 
tion ensures periodical textures. Using a sinusoidal activation 
function makes the training shorter and more stable for this 
task. During the generation, we optimise the material gloss level 
to reduce artefacts, like light reflection included in the texture. 


| 
Displacement 


Figure 4. Al Synesthetic Virtual Materials Generation Procedure 


Main Light 3 


Template 


For each training step, we start with generating the texture 
and the displacement along the normal for each vertex, then 
we produce a refined version of each render using the Stable 
Diffusion Img2Img pipeline conditioned on the text prompt. 
Finally, we compute the MSE (Mean Square Error) between 
the main render and the target image, then compare the ren- 
ders from the cameras sampled uniformly around the sphere 
and their refined version using a perceptual loss based on a 
pre-trained VGG19 net. 


This solution led to faster convergence and better quality 
than using a loss based on CLIP embeddings, moreover the 
refining procedure must be executed without the gradient 
computation, allowing to reduce the memory occupancy and 
increasing the training speed if compared to the SDS (Score 
Distillation Sampling) used in Magic3D, because it does not 
require backpropagation through the encoder network at the 
cost of having to decode the denoised latent vector. In this 
way, we produce textures with 712 pixels of resolution. To en- 
force a more coherent bump map, we follow the same pro- 
cedure with a uniform texture. To guarantee stability during 
the training, we must ensure semantic consistency between 
the refined images and the corresponding renderer. For this 
reason, we set a low strength (around 20%) and a not too high 
guidance scale (we used 12.5). 


VIRTUAL SKIN: CO-CREATING 3D MATERIALS WITH SYNESTHETIC ARTIFICIAL INTELLIGENCE 


Conclusion 

We aim to explore the possibilities of using Al for 3D virtual 
material generation, with the potential to arouse sufficient 
sensory and emotional stimuli. Hence, we have collected 
and designed multi-sensory material properties to gener- 
ate Al-generated 3D materials for virtual - and potentially 
hybrid - application to products and environments through 
User Experience Design and Color and Material Design 
(CMF) workflows. 


In the first of several on-going experiments, we have in- 
vestigated procedural approaches to collect sensory data 
to train our neural network model toward enabling its syn- 
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